Multi-purposed self-propelled device

ABSTRACT

A self-propelled device can include at least a wireless interface, a housing, a propulsion mechanism, and a camera. Using the camera, the self-propelled device can generate a video feed and transmit the video feed to a controller device via the wireless interface. The self-propelled device can receive an input from the controller device indicating an object or location in the video feed. In response to the input, the self-propelled device can initiate an autonomous mode to autonomously operate the propulsion mechanism to propel the self-propelled device towards the object or location indicated in the video feed.

RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.14/271,203, filed May 6, 2014 entitled “MULTI-PURPOSED SELF-PROPELLEDDEVICE,” which claims priority to U.S. Provisional Patent ApplicationSer. No. 61/820,109, filed May 6, 2013, entitled, “MULTI-PURPOSEDSELF-PROPELLED DEVICE,” filed May 6, 2013; and which is aContinuation-in-Part of U.S. patent application Ser. No. 14/035,841,filed Sep. 24, 2013, entitled “SELF-PROPELLED DEVICE WITH ACTIVELYENGAGED DRIVE SYSTEM,” now U.S. Pat. No. 9,193,404, issued Nov. 24,2015; which is a Continuation of U.S. patent application Ser. No.13/342,853, filed Jan. 3, 2012, entitled “SELF-PROPELLED DEVICE WITHACTIVELY ENGAGED DRIVE SYSTEM,”, now U.S. Pat. No. 8,571,781, issuedOct. 29, 2013; which claims benefit of priority to the following: U.S.Provisional Application No. 61/430,023, filed Jan. 5, 2011, entitled“METHOD AND SYSTEM FOR CONTROLLING A ROBOTIC DEVICE,”; U.S. ProvisionalApplication No. 61/430,083, filed Jan. 5, 2011, entitled “SYSTEM ANDMETHOD FOR ESTABLISHING 2-WAY COMMUNICATION FOR CONTROLLING A ROBOTICDEVICE,”; and U.S. Provisional Application No. 61/553,923, filed Oct.31, 2011, entitled “A SELF-PROPELLED DEVICE AND SYSTEM FOR CONTROLLINGSAME,”; all of the aforementioned priority applications being herebyincorporated by reference in their respective entireties for allpurposes.

TECHNICAL FIELD

Examples described herein relate to a multi-purpose remotelycontrollable self-propelled device.

BACKGROUND

Various types of remotely controllable devices exist. For examplehobbyists often operate remote controlled devices in the form of cars,trucks, airplanes and helicopters. Such devices typically receivecommands from a controller device, and alter movement (e.g., directionor velocity) based on the input. Some devices use software-basedcontrollers, which can be implemented in the form of an applicationrunning on a device such as a smart phone or a tablet.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure herein is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings in which likereference numerals refer to similar elements, and in which:

FIG. 1 is an example block diagram illustrating the components of aself-propelled device that is in the form of a spherical ball;

FIG. 2A is an example block diagram illustrating a self-propelleddevice;

FIG. 2B is an example block diagram illustrating a self-propelled devicewith an alternative biasing mechanism;

FIG. 2C is an example block diagram illustrating a self-propelled devicewith another alternative biasing mechanism to improve payload space;

FIG. 3 is an example block diagram illustrating a self-propelled deviceincluding an attached camera therein;

FIG. 4 is an example block diagram illustrating a self-propelled deviceincluding a payload;

FIG. 5 is an example schematic depiction of a self-propelled device anda controller device;

FIG. 6 illustrates an example technique for causing motion of aself-propelled spherical device;

FIG. 7 is an example block diagram depicting a sensor array and dataflow;

FIG. 8 illustrates an example system including a self-propelled deviceand a controller device to control and interact with the self-propelleddevice; and

FIG. 9 is an example flow chart illustrating a method of operating aself-propelled device.

DETAILED DESCRIPTION

A multi-purposed self-propelled device is provided which includes adrive system, a spherical housing, a biasing mechanism, and a payloadspace within the spherical housing. The biasing mechanism can includesingle extended spring to engage an inner surface of the sphericalhousing diametrically opposed to a contact surface engaged by the drivesystem. Alternatively, the biasing mechanism can include one or more(e.g., a pair) of portal axles each having a spring and contact elementto push against a respective contact point in the inner surface of thespherical housing. As such, the portal axles can produce a verticalforce that similarly actively forces the drive system to continuouslyengage the inner surface of the spherical housing in order to cause thespherical housing to move. The self-propelled device can be implementedto carry a payload or multiple payloads for various uses and activities.

The payload space can be utilized to carry any payload. For example, thepayload can include a camera, which can send images and/or a videostream from the self-propelled device to a controller device. Thecontroller device can be in the form of a smartphone, tablet, or anyother suitable operating device. Further, the spherical housing can betransparent to allow the camera to take images or real time video,thereby allowing a user of the controller device (e.g. smartphone) tocontrol the self-propelled device by viewing the video stream.

In still other variations, the payload space includes a single payloador multiple payloads for civilian or military use. Such payloads caninclude one or more of: a camera, an infrared sensor, a chemical sensor,a biological sensor, one or more explosives, a listening device, or anyother payload that can fit within the payload space.

For variations including a camera or other image capturing device, atouch a drive method is disclosed that provides the user with theability to dynamically control the self-propelled device by touchinglocation points on a live image or video feed. The method includesreceiving an image or live video feed from the self-propelled devicethat provides a field of view of the camera mounted within theself-propelled device. This video feed can be displayed on atouch-sensitive display of the controlled or mobile computing device.The method further includes receiving, on the touch-sensitive display, auser selection of a location point within the field of view of thecamera, and based on the user selection, generating a command signal tobe transmitted to the self-propelled device instructing theself-propelled device to maneuver to the location point.

Furthermore, the method can include determining a position of thelocation point in the field of view based on a first reference frame ofthe field of view as displayed on the controller device, and mapping theposition of the location point on a second reference frame correspondingto the location point relative to the self-propelled device. In sucharrangements, the command signal is generated to include instructionsfor maneuvering the self-propelled device based solely on the secondreference frame. Further still, for live video feed implementations, thecontroller device can generate the command signal dynamically inconjunction with receiving the live video feed.

System Overview

Referring to the drawings, FIG. 1 is an example block diagramillustrating the components of a self-propelled device that is in theform of a spherical ball. A multi-purposed self-propelled device 100 canbe operated to move under control of another device, such as a computingdevice (e.g. a smartphone, tablet, and/or remote control) operated by auser. The self-propelled device 100 can be configured with resourcesthat enable one or more of the following: (i) maintain self-awareness oforientation and/or position relative to an initial reference frame afterthe device initiates movement; (ii) process control inputprogrammatically, so as to enable a diverse range of program-specificresponses to different control inputs; (iii) enable another device tocontrol its movement using software or programming logic that iscommunicative with programming logic on the self-propelled device;and/or (iv) generate an output response for its movement and state thatit is software interpretable by the control device.

For example, the multi-purposed self-propelled device 100 can includeseveral interconnected subsystems and modules. A processor 114 executesprogrammatic instructions from a program memory 104. The instructionsstored in the program memory 104 can be changed, for example to addfeatures, correct flaws, or modify behavior. In some examples, theprogram memory 104 stores programming instructions that arecommunicative or otherwise operable with software executing on acomputing device. The processor 114 can be configured to executedifferent programs of programming instructions, in order to alter themanner in which the self-propelled device 100 interprets or otherwiseresponds to control input from another computing device.

A wireless communication module 110, in conjunction with a communicationtransducer 102, can serve to exchange data between processor 114 andother external devices. The data exchanges, for example, can providecommunications, provide control, provide logical instructions, stateinformation, and/or provide updates for the program memory 104. In someexamples, the processor 114 can generate output corresponding to stateand/or position information, that can then be communicated to thecontroller device via a wireless communication port. The mobility of thedevice makes wired connections undesirable. Therefore, the term“connection” can be understood to describe a logical link made without aphysical attachment to the self-propelled device 100.

In some examples, the wireless communication module 110 can implement aBLUETOOTH communications protocol and the transducer 102 can be anantenna suitable for transmission and reception of BLUETOOTH signals. Asan addition or alternative, the wireless communication module 110 canimplement a Wi-Fi communications protocol and the transducer can be anantenna suitable for transmission and reception of Wi-Fi signals. Insuch examples, the self-propelled device 100 can be controlled by acontroller device via BLUETOOTH and/or Wi-Fi signals. Other wirelesscommunication mediums and protocols can also be used in alternativeimplementations.

Sensors 112 can provide information about the surrounding environmentand condition to the processor 114. In variations, the sensors 112include inertial measurement devices, including a three-axis gyroscope,a three-axis accelerometer, and a three-axis magnetometer. Furthermore,the sensors 112 can provide input to enable the processor 114 tomaintain awareness of the device's orientation and/or position relativeto the initial reference frame after the device initiates movement. Thesensors 112 can include instruments for detecting light, temperature,humidity, or measuring chemical concentrations or radioactivity.

State/variable memory 106 can store information about the state of thedevice, including, for example, position, orientation, rates of rotationand translation in each axis. The state/variable memory 106 can alsostore information corresponding to an initial reference frame of thedevice upon, for example, the device being put in use (e.g., the devicebeing activated), as well as position and orientation information oncethe device is in use. In this manner, the self-propelled device 100 canutilize information of the state/variable memory 106 in order tomaintain position and orientation information of the self-propelleddevice 100 once the device is in operation.

A clock 108 can provide timing information to the processor 114. Forexample, the clock 108 can provide a time base for measuring intervalsand rates of change. Furthermore, the clock 108 can provide day, date,year, time, and alarm functions. Further still, the clock 108 can allowthe self-propelled device 100 to provide an alarm or alert at pre-settimes.

An expansion port 120 can provide a connection for addition ofaccessories or devices. Expansion port 120 provides for futureexpansion, as well as flexibility to add options or enhancements. Forexample, expansion port 120 can be used to add peripherals, sensors,processing hardware, storage, displays, or actuators to theself-propelled device 100.

As an addition or alternative, the expansion port 120 can provide aninterface capable of communicating with a suitably configured componentusing analog or digital signals. The expansion port 120 can provideelectrical interfaces and protocols that are standard or well-known. Theexpansion port 120 can further implement an optical interface. Forexample, interfaces appropriate for expansion port 120 include aUniversal Serial Bus (USB), Inter-Integrated Circuit Bus (I2C), SerialPeripheral Interface (SPI), or ETHERNET.

A display 118 presents information to outside devices or persons. Thedisplay 118 can present information in a variety of forms. In variousexamples, the display 118 can produce light in colors and patterns,sound, vibration, or combinations of sensory stimuli. In some examples,the display 118 can operate in conjunction with actuators 126 tocommunicate information by physical movements of the self-propelleddevice 100. For example, the self-propelled device 100 can be made toemulate a human head nod or shake to communicate “yes” or “no.”

As an addition or alternative, the display 118 can be configured to emitlight, either in the visible or invisible range. Invisible light in theinfrared or ultraviolet range can be useful, for example, to sendinformation invisible to human senses but available to specializeddetectors. Furthermore, the display 118 can include an array of LightEmitting Diodes (LEDs) emitting various light frequencies, arranged suchthat their relative intensity can be variable and the light emitted canbe blended to form color mixtures.

As an addition or alternative, the display 118 includes an LED arraycomprising several LEDs, each emitting a human-visible primary color.The processor 114 can vary the relative intensity of each LED to producea wide range of colors. Primary colors of light are those wherein a fewcolors can be blended in different amounts to produce a wide gamut ofapparent colors. Many sets of primary colors of light are known,including for example red/green/blue, red/green/blue/white, andred/green/blue/amber. For example, red, green, and blue LEDs togethercomprise a usable set of three available primary-color devicescomprising the display 118. In variations, other sets of primary colorsand white LEDs can be used. In many implementations, the display 118 caninclude one or more LEDs used to indicate a reference point on theself-propelled device 100 for alignment.

An energy storage unit 124 stores energy for operating the electronicsand electromechanical components of the self-propelled device 100. Forexample, the energy storage unit 124 can be a rechargeable battery. Aninductive charge port 128 can allow for recharging the energy storageunit 124 without a wired electrical connection. Furthermore, inductivecharge port 128 can receive magnetic energy and convert it to electricalenergy to recharge the energy storage unit 124. As an addition oralternative, the inductive charge port 128 can provide a wirelesscommunication interface with an external charging device. Further, aplug in charging means can be included as an addition or alternative tothe inductive charge port 128.

A deep sleep sensor 122 can be included to place the self-propelleddevice 100 into a very low power or “deep sleep” mode where most of theelectronic devices use no power. This may useful for long-term storage,shipping, and/or certain implementations of the self-propelled device100 that require such a state. For example, the self-propelled device100 can be implemented to carry a payload, such as a camera, a motionsensor, an infrared sensor and/or a chemical or biological sensor. Insuch variations, the self-propelled device 100 can sit dormant in thedeep sleep mode until a trigger, such as an event setting off the cameraor one or more sensors, automatically activates the self-propelleddevice 100.

In variations, the sensors 122 can sense objects, events, incidents,matter, and/or phenomena through the spherical housing of self-propelleddevice 100 without a wired connection. As an addition or alternative,the deep sleep sensor 122 may be a Hall Effect sensor mounted in amanner such that an external magnet can be applied on the self-propelleddevice 100 to activate the deep sleep mode.

The drive system actuators 126 can convert electrical energy intomechanical energy for various uses. A primary use of the actuators 126can be to propel and steer the self-propelled device 100. Movement andsteering actuators can also be referred to as a drive system or tractionsystem. The drive system causes rotational and translational movement ofthe self-propelled device 100 under control of the processor 114.Examples of the actuators 126 can include, for example, wheels, motors,solenoids, propellers, paddle wheels, and pendulums.

For example, the drive system actuators 126 can include a set of twoparallel wheels, each mounted to an axle connected to independentlyvariable-speed motors through a reduction gear system. In such examples,the operation of the two independently operated drive motors can becontrolled by processor 114.

However, the actuators 126 may produce a variety of movements inaddition to rotating and translating self-propelled device 100. Forexample, the actuators 126 can cause self-propelled device 100 toexecute communicative movements, including emulation of human gestures,for example, head nodding, shaking, trembling, spinning or flipping. Invariations, the processor coordinates the actuators 126 with the display118. Similarly, the processor 114 can provide signals to the actuators126 and the display 118 to cause the self-propelled device 100 to spinor tremble and simultaneously emit patterns of colored light. As anaddition or alternative, the self-propelled device 100 can emit light orsound patterns synchronized with movements.

The self-propelled device 100 can be used as a controller for othernetwork-connected devices. The self-propelled device 100 can containsensors and wireless communication capabilities, so that it may performa controller role for other devices. For example, the self-propelleddevice 100 may be held in the hand and used to sense gestures,movements, rotations, combination inputs and the like.

Mechanical Design

FIG. 2A is an example block diagram illustrating a self-propelleddevice. The self-propelled device 200 can be multi-purposed and caninclude a spherical housing 202, a drive system 201, a biasing mechanism215, and a payload space 228. For example, the self-propelled device 200may be in the form of a robotic, spherical ball. The self-propelleddevice 200 can be of a size and weight allowing it to be easily grasped,lifted, and carried in an adult human hand. In variations, theself-propelled device 200 may be larger or smaller, and/or customizedfor a particular payload. For example, the self-propelled device 200 maybe manufactured to include a transparent spherical housing, and a cameramay be included within the payload space to provide a video stream tothe controller device, such as for example as smartphone or tablet, asshown by an example of FIG. 3.

Referring still to FIG. 2A, the self-propelled device 200 includes anouter spherical shell (or housing) 202 that makes contact with anexternal surface as the device rolls. In addition, the self-propelleddevice 200 includes an inner surface 204 of the spherical housing 202,wherein wheels 218 and 220 included as components of the drive system201 make contact with the inner surface 204 to cause the self-propelleddevice 200 to move. Additionally, the self-propelled device 200 caninclude several mechanical and electronic components enclosed within thespherical housing 202.

The spherical housing 202 can be at least partially composed of one ormore materials that allow for the transmission of signals used forwireless communication, and yet can be impervious to moisture and dirt.The spherical housing 202 material can be durable, washable, and/orshatter resistant. The spherical housing 202 can also be structured toenable transmission of light and can be textured to diffuse the light.

In variations, the spherical housing 202 can be made of a sealedpolycarbonate plastic. In similar variations, at least one of thespherical housing 202, or the inner surface 204, can be textured todiffuse light. As an addition or an alternative, the spherical housing202 can comprise two hemispherical shells with an associated attachmentmechanism, such that the spherical housing 202 can be opened to allowaccess to the internal electronic and mechanical components. In similarvariations, the spherical housing 202 can include automatic openingmeans configured to open the housing upon a command prompt by a userusing the controller device. As an addition or alternative, thespherical housing 202 can be opened to deposit a payload placed in apayload space 228, and then subsequently closed such that the user cancontinue maneuvering the self-propelled device 200. The sphericalhousing 202 can further include a treaded outer surface, or an outersurface that includes knobs and/or nodules for traction.

Several electronic and mechanical components can be positioned insidethe spherical housing 202 for enabling processing, wirelesscommunication, propulsion, and other functions. Among the components, adrive system 201 can be included to enable the device to propel itself.The drive system 201 can be coupled to processing resources and othercontrol mechanisms as described above with reference to FIG. 1.Referring still to FIG. 2A, a carrier 214 can be included to serve as anattachment point and support for electronic components of theself-propelled device 200. The drive system 201, energy storage unit216, carrier 214, biasing mechanism 215, other components such asdescribed above with reference to FIG. 1, and any payload items that canultimately occupy a payload space 228, cannot be rigidly mounted to thespherical housing 202. Instead, wheels 218, 220 included in the drivesystem 201 can be in frictional contact with the inner surface 204 ofthe spherical housing 202, and can function to drive the sphericalhousing 202 by the action of the actuators.

The carrier 214 can be in mechanical and electrical contact with theenergy storage unit 216. The energy storage unit 216 can provide areservoir of energy to power the device and electronics and can bereplenished via the inductive charge port 226. For example, the energystorage unit 216 can a rechargeable battery and can be composed oflithium-polymer cells.

The carrier 214 can provide a mounting location for most of the internalcomponents, including printed circuit boards for electronic assemblies,the sensor array, one or more antenna, and one or more connectors, aswell as providing a mechanical attachment point for internal components.Further, the carrier 214 can provide a base for payloads being placedwithin the self-propelled device 200. In this manner, the top surface ofthe carrier 214 can also be the floor of the payload space 228. However,other configurations of the payload space 228 are contemplated, such asfor example, a payload space 228 specially formed for a camera, or onethat includes compartments for carrying multiple items.

The drive system 201 includes motors 222, 224 and the wheels 218, 220.The motors 222, 224 connect to the wheels 218, 220 independently throughan associated shaft, axle, and gear drive (not shown). The wheels 218,220 can be in physical contact with inner surface 204. The points wherewheels 218, 220 contact the inner surface 204 are an essential part ofthe drive mechanism of the self-propelled device 200, and so arepreferably coated with a material to increase friction and reduceslippage. For example, the circumference of the wheels 218, 220 can becoated with silicone rubber tires.

The biasing mechanism 215 can be included to actively force the wheels218, 220 against the inner surface 204. As an example, a spring 212 andspring end 210 can comprise the biasing mechanism 215. Morespecifically, the spring 212 and the spring end 210 can be positioned tocontact the inner surface 204 at a point diametrically opposed to thewheels 218, 220. The spring 212 and the spring end 210 can providecontact force to reduce or substantially eliminate slippage of thewheels 218, 220. The spring 212 is selected to provide a small forcepushing wheels 218, 220, and the spring end 210 evenly against innersurface 204.

The spring end 210 provides near-frictionless contact with the innersurface 204. As such, the spring end 210 comprises a rounded orquasi-rounded surface configured to slide along the inner surface 204 asthe self-propelled device 200 is driven by the drive system 201.Additional means of providing near-frictionless contact can be included.In some implementations, the spring end 210 can include one or morebearings to further reduce friction at the contact point where thespring end 210 moves along the inner surface 204. Furthermore, thespring 212 and the spring end 210 can be composed of a non-magneticmaterial to avoid interference with sensitive magnetic sensors.

The payload space 228 can be incorporated to carry a single payload ormultiple payloads for civilian and/or military activities. The payloadspace 228 can be arranged to carry any number of payloads, including,for example, one or more of: a camera for surveillance or remote videostream, an infrared sensor, a chemical or biological sensor to detectharmful agents, an explosive or flash bang that can be maneuvered intoplace and detonated, and/or a listening device.

FIG. 2B is an example block diagram illustrating a self-propelled devicewith an alternative biasing mechanism 217. Referring to FIG. 2B, thebiasing mechanism 217 can incorporate a chassis 237 which can includeone or more additional wheels 234, 236 to increase or reconfigure thepayload space 228. The biasing mechanism 217 can include a center post238 configured to exert a force against the chassis 237 which can, inturn, press the additional wheels 234, 236 against inner surface 204 ofthe spherical housing 202. The center post 238 may or may not include aspring 240 to aid in the biasing. Alternatively, the chassis 237 may beplaced proximate to the drive system 201 such that the wheels 218, 220,234, and 236 contact the inner surface 204 in the same hemisphere, asopposed to being diametrically opposite. As an addition or alternative,the chassis 237 may be positioned such that the center post 238 forms anangle, wherein the additional wheels 234, 236 contact the inner surface204 slightly above the equator (in a reference frame where the drivesystem 201 is positioned at the very bottom of the self-propelled device200). These arrangements, and similar arrangements, may be included to,for example, maximize or optimize the payload space 228.

FIG. 2C is an example block diagram illustrating a self-propelled devicewith another alternative biasing mechanism to improve payload space 250.For example, an independent biasing mechanism 252 can be includedconfigured to actively force the drive system wheels 218, 220 againstthe inner surface 204, similar to the biasing mechanisms discussed abovewith respect to FIGS. 2A and 2B. The independent biasing mechanism 252is comprised of two or more separate portal axles 258, 260. The portalaxles 258, 260 may include springs to press respective wheels 254, 256against the inner surface 204 with a force vector having a verticalvalue. The vertical force from the wheels 254, 256 pressing against theinner surface 204 in turn, actively forces the drive system 201 and itsrespective wheels 254, 256 against the inner surface 204 as well,thereby providing sufficient force for the drive system 201 to cause theself-propelled device 200 to move.

As shown, the independent biasing mechanism 252 with its two or moreportal axles 258, 260 allows for significantly increased payload space250, whereby any number of different types or sizes of payload (e.g. acamera, infrared sensor, chemical or biological sensor, one or moreexplosives, a listening device, etc.) can be positioned within the space250. By removing the biasing mechanism configurations as shown byexamples of FIGS. 2A and 2B, and replacing them with an independentbiasing mechanism 252 with independent portal axles 258, 260, theinterior of the self-propelled device 200 can be cleared to provide aspacious interior. The portal axles 258, 260 comprising the independentbiasing mechanism 252 can be mounted directly onto the carrier 214. Thesprings corresponding to the portal axles 258, 260 may be in the form oftorsion springs which force the wheels 254, 256 against the innersurface 204. As an addition or alternative, the springs may be comprisedof one or more of a compression spring, clock spring, or tension spring.Alternatively, the portal axles 258, 260 can be mounted in such a mannerin which no springs are included to maintain a force pressing the drivesystem 201 and wheels 218, 220 against the inner surface 204, allowingsufficient traction to cause the self-propelled device 200 to move.

FIG. 3 is an example block diagram illustrating a self-propelled deviceincluding a camera. The multi-purpose self-propelled device 300 caninclude a camera 302 mounted within the payload space 310. For example,the spherical housing 304 is transparent to allow the camera 302 to viewoutside the self-propelled device 300. The camera 302 can be mounted onthe biasing mechanism, or otherwise mounted within the self-propelleddevice 300 in order to maintain a substantially level orientation. Forexample, the camera 302 can be mounted to the carrier 314 to achieve thesame.

The camera 302 can be mounted on a gyroscope or other stabilizing meansso that the camera's orientation can be more substantially level with anoutside reference frame. For example, the stabilizing means (not shown)can be mounted on the biasing mechanism 315 and/or the carrier 314 inorder to achieve further stability and to further maintain orientation.For self-propelled devices 300 having an independent biasing mechanismusing portal axles as discussed with respect to FIG. 2C, the camera 302can be mounted to the carrier 314.

As an addition or alternative, the camera 302 can further be coupled tothe wireless communication module. The camera 302 can providephotographs and/or a video stream to the controller device 306 by way ofa wireless link 308. In similar arrangements, the self-propelled devicecan be remotely controlled by a user using the controller device 306 viathe wireless link 308. As implemented, the self-propelled device 300 canbe controlled utilizing the video stream, and therefore, the user neednot be in visual sight of the self-propelled device 300 in order tomaintain effective operation.

The camera 302 can be any photo or video recording device. The camera302 can itself contain wireless communication means, such that it maydirectly link with the controller device 306. Alternatively, the camera302 can be coupled to the processor and the video stream may be fed tothe controller device 306 via the wireless module included in the systemas described above with respect to FIG. 1.

The controller device 306 can be a remote control, smartphone, tablet,or a customized device capable of storing and executing applications orinstructions to operate the self-propelled device 300. For example, thecontroller device 306 can be in the form of a smartphone or tablet, andoperation of the self-propelled device can be executed by way of anapplication stored in the smartphone or tablet. Alternatively, thecontroller device 306 and self-propelled device 300 can be manufacturedas a combined system specially manufactured for one or more purposes.For example, the combined system of the self-propelled device 300 andthe controller device 306 can be used specifically for surveillance.Further, an application of the controller device 306 can include a“touch and drive” feature, wherein the user can touch the display of thecontroller device 306 corresponding to a selected location, and causedthe self-propelled device 300 to maneuver to that location. The “touchand drive” feature can further allow the user to select a location onthe display of the controller device 306, where the selected locationcorresponds to the same location on the video stream. In other words,the user can “touch” a location on the video stream from theself-propelled device 300, and the self-propelled device 300 willautomatically maneuver itself to that location.

FIG. 4 is an example block diagram illustrating a self-propelled deviceincluding a payload. Referring to FIG. 4, the self-propelled deviceincludes a payload space 406 to carry and transport specific items orpayload 404 for various activities. Such items can include one or moreof: an infrared sensor, a chemical or biological sensor, an explosive orflash bang, a listening device, other mechanical devices, and the like.Any one or more of these items can further be included in variationsincluding a camera as described above with reference to FIG. 3.

As shown in an example of FIG. 4, the self-propelled device 400 includesa payload space 406 for carrying a payload 404. The self-propelleddevice 400 can be constructed in manner in which it can be thrown by auser, and then controlled via the controller device 412. A protectivecover 402 can be included to dampen shock when, for example, theself-propelled device 400 is thrown by a user. The protective cover 402can be removable and/or can be configured to break away from theself-propelled device 400 during periods of extreme shock. In suchexamples, the self-propelled device 400 can break free of the protectivecover 402 and proceed with its intended purpose.

The protective cover 402 can be included to seal or waterproof theself-propelled device 400. For example, the self-propelled device 400can be further configured to float or sink depending on its intendedapplication. Furthermore, the self-propelled device 400 can float whendeposited into water in order to be easily retrievable.

Examples as shown in FIG. 4 can be manufactured for a singular purpose,or can be configured to be loaded and unloaded multiple times. Thelatter can include a spherical housing 408 and/or a protective cover 402that can be readily opened or closed by a user. Single purpose examplescan be manufactured to be permanently sealed, and the payload 404 ofsuch arrangements can be selected for the singular purpose. For example,the payload 404 can be an explosive, where the self-propelled device 400is utilized for a single use. As implemented, the self-propelled device400 can be maneuvered to a selected location and then detonated,effectively destroying the self-propelled device 400.

Alternatively, the self-propelled device 400 can be configured to openupon a command prompt by a user and deposit its payload 404 at aselected location. The user can then cause the spherical housing 408 toclose using the controller device 412 and proceed to continuemaneuvering the self-propelled device 400. Furthermore, theself-propelled device 400 and controller device 412 can include the“touch and drive” feature as described above with respect to FIG. 4.

FIG. 5 is an example schematic depiction of a self-propelled device 514and a controller device 508. More specifically, the self-propelleddevice 514 can be controlled in its movement by programming logic and/orcontrols that can originate from the controller device 508. Theself-propelled device 514 is capable of movement under control of thecontroller device 508, which can be operated by a user 502. Thecontroller device 508 can wirelessly communicate control data to theself-propelled device 514 using a standard or proprietary wirelesscommunication protocol. In variations, the self-propelled device 514 canbe at least partially self-controlled, utilizing sensors and internalprogramming logic to control the parameters of its movement (e.g.,velocity, direction, etc.). Still further, the self-propelled device 514can communicate data relating to the device's position and/or movementparameters for the purpose of generating or alternating content on thecontroller device 508. In additional variations, self-propelled device514 can control aspects of the controller device 508 by way of itsmovements and/or internal programming logic.

As described herein, the self-propelled device 514 can have multiplemodes of operation, including those of operation in which the device iscontrolled by the controller device 508, is a controller for anotherdevice (e.g., another self-propelled device or the controller device508), and/or is partially or wholly self-autonomous.

Additionally, the self-propelled device 514 and the controller device508 can share a computing platform on which programming logic is shared,in order to enable, among other features, functionality that includes:(i) enabling the user 502 to operate the controller device 508 togenerate multiple inputs, including simple directional input, commandinput, gesture input, motion or other sensory input, voice input orcombinations thereof; (ii) enabling the self-propelled device 514 tointerpret inputs received from the controller device 508 as a command orset of commands; and/or (iii) enabling the self-propelled device 514 tocommunicate data regarding that device's position, movement and/or statein order to effect a state on the controller device 508 (e.g., displaystate, such as content corresponding to a controller-user interface).Furthermore, the self-propelled device 514 can include a programmaticinterface that facilitates additional programming logic and/orinstructions to use the device. The controller device 508 can executeprogramming that is communicative with the programming logic on theself-propelled device 514.

Accordingly, the self-propelled device 514 can include an actuator ordrive mechanism causing motion or directional movement. Theself-propelled device 514 can be referred to by a number of relatedterms and phrases, including controlled device, robot, robotic device,remote device, autonomous device, and remote-controlled device. Invariations, the self-propelled device 514 can be structured to move andbe controlled in various media. For example, the self-propelled device514 can be configured for movement in media such as on flat surfaces,sandy surfaces, or rocky surfaces.

The self-propelled device 514 can be implemented in various forms. Forexample, the self-propelled device 514 can correspond to a sphericalobject that can roll and/or perform other movements such as spinning. Invariations, the self-propelled device 514 can correspond to aradio-controlled aircraft, such as an airplane, helicopter, hovercraftor balloon. In other variations, the self-propelled device 514 cancorrespond to a radio controlled watercraft, such as a boat orsubmarine. Numerous other variations can also be implemented, such asthose in which the self-propelled device 514 is a robot.

Furthermore, the self-propelled device 514 can include a sealed hollowhousing, roughly spherical in shape, capable of directional movement byaction of actuators inside the enclosing envelope.

Referring still to FIG. 5, the self-propelled device 514 can beconfigured to communicate with the controller device 508 using networkcommunication links 510 and 512. Link 510 can transfer data from thecontroller device 508 to the self-propelled device 514. Link 512 cantransfer data from the self-propelled device 514 to the controllerdevice 508. Links 510 and 512 are shown as separate unidirectional linksfor illustration. As an example, a single bi-directional communicationlink performs communication in both directions. Link 510 and link 512are not necessarily identical in type, bandwidth or capability. Forexample, communication link 510 from controller device 508 toself-propelled device 514 is often capable of a higher communicationrate and bandwidth compared to link 512. In some situations, only onelink 510 or 512 is established. In such examples, communication isunidirectional.

The controller device 508 can correspond to any device comprising atleast a processor and communication capability suitable for establishingat least unidirectional communications with the self-propelled device514. Examples of such devices include, without limitation: mobilecomputing devices (e.g., multifunctional messaging/voice communicationdevices such as smart phones), tablet computers, portable communicationdevices and personal computers. For example, the controller device 508is an IPHONE available from APPLE COMPUTER, INC. of Cupertino, Calif. Inanother example, the controller device 508 is an IPAD tablet computer,also from APPLE COMPUTER. In still other examples, the controller device508 is any of the handheld computing and communication appliancesexecuting the ANDROID operating system from GOOGLE, INC.

Elsewise, the controller device 508 can be a personal computer, ineither a laptop or desktop configuration. For example, the controllerdevice 508 can be a multi-purpose computing platform running theMICROSOFT WINDOWS operating system, or the LINUX operating system, orthe APPLE OS/X operating system, configured with an appropriateapplication program to communicate with the self-propelled device 514.

In variations, the controller device 508 can be a specialized devicededicated for enabling the user 502 to control and interact with theself-propelled device 514.

Furthermore, multiple types of controller devices 508 can be usedinterchangeably to communicate with the self-propelled device 514.Additionally or as an alternative, the self-propelled device 514 can becapable of communicating and/or being controlled by multiple devices(e.g., concurrently or one at a time). For example, the self-propelleddevice 514 can be capable of being linked with an IPHONE in one sessionand with an ANDROID device in a later session, without modification ofthe device 514.

Accordingly, the user 502 can interact with the self-propelled device514 via the controller device 508, in order to control theself-propelled device 514 and/or to receive feedback or interaction onthe controller device 508 from the self-propelled device 514.Furthermore, the user 502 can specify a user input 504 through variousmechanisms that are provided with the controller device 508. Examples ofsuch inputs include text entry, voice command, touching a sensingsurface or screen, physical manipulations, gestures, taps, shaking andcombinations of the above.

The user 502 can interact with the controller device 508 in order toreceive feedback 506. The feedback 506 can be generated on thecontroller device 508 in response to user input. As an alternative oraddition, the feedback 506 can also be based on data communicated fromthe self-propelled device 514 to the controller device 508, regarding,for example, the self-propelled device's 514 position or state. Withoutlimitation, examples of feedback 506 include text display, graphicaldisplay, sound, music, tonal patterns, modulation of color or intensityof light, haptic, vibrational or tactile stimulation. The feedback 506can be combined with input that is generated on the controller device508. For example, the controller device 508 can output content that canbe modified to reflect position or state information communicated fromthe self-propelled device 514.

As an example, the controller device 508 and/or the self-propelleddevice 514 can be configured such that user input 504 and feedback 506maximize usability and accessibility for the user 502, who has limitedsensing, thinking, perception, motor or other abilities. This allowsusers with handicaps or special needs to operate the system 500 asdescribed.

A configuration as illustrated in FIG. 5, can be only one of an almostunlimited number of possible configurations of networks including aself-propelled device 514 with communication connections. Furthermore,while numerous variations described herein provide for a user to operateor otherwise directly interface with the controller device 508 in orderto control and/or interact with a self-propelled device, variationsdescribed encompass enabling the user to directly control or interactwith the self-propelled device 514 without use of an intermediary devicesuch as the controller device 508.

FIG. 6 illustrates an example technique for causing motion of aself-propelled spherical device 600. As shown by an example of FIG. 6,the device 600 has a center of rotation 602 and a center of mass 606,the device 600 being contact with planar surface 612. The drivemechanism for the robotic device 600 comprises twoindependently-controlled wheeled actuators 608 in contact with the innersurface of the enclosing spherical envelope of device 600. Also shown issensor platform 604. Several components of device 600 are not shown inFIG. 6 for simplicity of illustration.

To achieve continuous motion at a constant velocity, the displacement ofcenter of mass 606 relative to center of rotation 602 can be maintainedby action of wheeled actuators 608. The displacement of the center ofmass 606 relative to the center of rotation 602 is difficult to measure,thus it is difficult to obtain feedback for a closed-loop controller tomaintain constant velocity. However, the displacement is proportional tothe angle 610 between sensor platform 604 and surface 612. The angle 610can be sensed or estimated from a variety of sensor inputs, as describedherein. Therefore, the speed controller for robotic device 600 can beimplemented to use angle 610 to regulate speed for wheeled actuators 608causing the device 600 to move at a constant speed across the surface612. The speed controller can determine the desired angle 610 to producethe desired speed, and the desired angle set point can be provided as aninput to a closed loop controller regulating the drive mechanism.

FIG. 6 illustrates use of angle measurement for speed control. However,the technique can further be extended to provide control of turns androtations, with feedback of appropriate sensed angles and angular rates.

It can be seen from the foregoing discussion that knowledge of theorientation angles is useful, in variations, for control of theself-propelled device 600. Measuring the orientation of the device canalso be useful for navigation and alignment with other devices.

FIG. 7 is an example block diagram depicting a sensor array and dataflow. Referring to FIG. 7, the sensor array 712 can include a set ofsensors for providing information to the self-propelled device,including for example, its position, orientation, rates of translation,rotation, and acceleration. Many other sensors can be included to meetrequirements in various examples.

In variations, the sensor array 712 can include a three-axis gyroscopicsensor 702, a three-axis accelerometer sensor 704, and a three-axismagnetometer sensor 706. In certain variations, a receiver for a GlobalPositioning System (GPS) is included 710. However, GPS signals aretypically unavailable indoors, so the GPS receiver can be omitted.

Due to limitations in size and cost, the sensors in sensor array 712 canbe miniaturized devices employing micro-electro-mechanical (MEMS)technology. The data from these sensors can require filtering andprocessing to produce accurate state estimates 716. Various algorithmscan be employed in sensor fusion and state estimator 714. Thesealgorithms can be executed by the processor on the self-propelleddevice.

Those familiar with the art will understand that the signals from sensorin sensor array 712 can be imperfect and distorted by noise,interference, and the limited capability of inexpensive sensors.However, the sensors can also provide redundant information, so thatapplication of a suitable sensor fusion and state estimator process 714can provide an adequate state estimation 716 of the true state of theself-propelled device.

For example, in many situations, magnetometer data is distorted by straymagnetic fields and ferrous metals in the vicinity. Sensor fusion andstate estimator 714 can be configured to reject bad or suspectmagnetometer data and rely on the remaining sensors in estimating thestate 716 of the self-propelled device. In some examples, particularmovements of the self-propelled device can be used to improve sensordata for desired purposes. For example, it can be useful to rotateself-propelled device through an entire 360 degree heading sweep whilemonitoring magnetometer data, to map local magnetic fields. Since thefields are usually relatively invariant over a short period of time, thelocal field measurement is repeatable and therefore useful, even ifdistorted.

Architecture

FIG. 8 illustrates an example system including a self-propelled device810 and a controller device 850 that controls and interacts with theself-propelled device 810. The self-propelled device 810 can beconstructed using hardware resources such as described by examples ofFIG. 1. Accordingly, the self-propelled device 810 can be a sphericalobject such as described by examples with respect to FIGS. 2-4. Thecontroller device 850 can be a multifunctional device, such as a mobilecomputing device (e.g., smart phone), tablet, or personal computer.Alternatively, the controller device 850 can correspond to a specializeddevice that is dedicated to controlling and communicating with theself-propelled device 810.

The self-propelled device 810 can execute one or more programs 816stored in a program library 820. Each program 816 in the program library820 can include instructions or rules for operating the device,including instructions for how the device is to respond to specificconditions, how the device is to respond to a control input 813 (e.g.,user input entered on the controller device 850), and/or the mode ofoperation that the device is to implement (e.g., controlled mode, versusautonomous, etc.).

The program library 820 can also maintain an instruction set that can beshared by multiple programs, including instructions that enable someuser inputs to be interpreted in a common manner. An application programinterface (API) 830 can be implemented on the device 810 to enableprograms to access a library of functions and resources of the device.For example, the API 830 can include functions that can be used withprograms to implement motor control (e.g., speed or direction), statetransition, sensor device interpretation, and/or wirelesscommunications.

In certain implementations, the device 810 can receive programs andprogramming instructions wirelessly through the use of a wirelesscommunication port 812. In variations, the device 810 can receiveprograms and programming instructions 882 from external sources 880 viaother ports, such as an expansion port 120 (see FIG. 1). The programmingresources can originate from, for example, a media provided to the userof the device (e.g., SD card), a network resource or website whereprograms can be downloaded, and/or programs and/or instruction setscommunicated via the wireless communication port 812 from the controllerdevice 850. In several implementations, the controller device 850 can beprogrammatically configured to interact and/or control theself-propelled device 810 with software. Once configured, the controllerdevice 850 can communicate instructions coinciding with its programmaticconfiguration to the self-propelled device 810. For example, thecontroller device 850 can download an application for controlling orinteracting with the self-propelled device 810. The application can bedownloaded from, for example, a network (e.g., from an App Store), orfrom a website, using wireless communication capabilities inherent inthe controller device 850 (e.g., cellular capabilities, Wi-Ficapabilities, etc.). The application that is downloaded by thecontroller device 850 can include an instruction set that can becommunicated to the self-propelled device 810.

The controller device 850 can execute a program 856 that is specializedor otherwise specific to communicating or interacting with, and/orcontrolling the self-propelled device 810. In variations, the program856 that executes on the controller device 850 can include a counterpartprogram 816A that can execute on the self-propelled device 810. Theprograms 856, 816A can execute as a shared platform or system. Forexample, as described below, the program 856 operating on the controllerdevice 850 can cooperate with the counterpart runtime program 816A togenerate input for the self-propelled device 810, and to generate outputon the controller device 850 based on a data signal from theself-propelled device 810. For example, the program 856 can generate auser interface 860 that (i) prompts or provides guidance for the user toprovide input that is interpretable on the self-propelled device 810 asa result of the counterpart runtime program 816A, resulting in someexpected outcome from the self-propelled device 810; and (ii) receivesfeedback 818 from the self-propelled device 810 in a manner that affectsthe content that is output by the program 856 operating on thecontroller device 850. In the latter case, for example,computer-generated content can be altered based on positioning ormovement of the self-propelled device 810.

More specifically, on the controller device 850, the program 856 canprovide a user interface 860, including logic 862 for prompting and/orinterpreting user input on the controller device. Various forms of inputcan be entered on the controller device 850, including, for example,user interaction with mechanical switches or buttons, touchscreen input,audio input, gesture input, or movements of the device in a particularmanner.

Accordingly, the program 856 can be configured to utilize an inherentapplication program interface on the controller device 850, to utilizethe various resources of the device to receive and process input. Manyexisting multifunctional or general purpose computing devices (e.g.,smart phones or tablets) can be configured to detect various kinds ofinput, including touchscreen input (e.g., multi-touch input or gestureinput), optical input (e.g., camera image sensing input), audio inputand device movement input (e.g., shaking or moving the entire device).The user interface 860 can include logic 862 to prompt the user forspecific kinds of input (e.g., include visual markers where a usershould place fingers, instruct the user or provide the user with thevisual and/or audio prompt to move the device, etc.), and to interpretthe input into control information that is signaled to theself-propelled device 810.

In some implementations, the input generated on the controller device850 can be interpreted as a command and then signaled to theself-propelled device 810. In other implementations, the input enteredon the controller device 850 can be interpreted as a command byprogrammatic resources on the self-propelled device 810. By interpretinguser input in the form of commands, the self-propelled device 810 canrespond to user input in a manner that is intelligent and configurable.For example, the self-propelled device 810 can interpret user input thatis otherwise directional in nature in a manner that is not directional.For example, a user can enter gesture input corresponding to adirection, in order to have the self-propelled device 810 move in amanner that is different than the inherent direction in the user input.For example, a user can enter a leftward gesture, which the device caninterpret (based on the runtime program 816A) as a command to stop,spin, return home, or alter illumination output, etc.

The user interface 860 can also include output logic 864 forinterpreting data received from the self-propelled device 810. As such,the self-propelled device 810 can communicate information, such as stateinformation and/or position information (e.g., such as after when thedevice moves) to the controller device 850. In one implementation, thecommunication from the self-propelled device 810 to the controllerdevice 850 can be in response to a command interpreted from user inputon the controller device 850. In another implementation, thecommunication from the self-propelled device 810 can be in the form ofcontinuous feedback generated as result of the device's continuousmovement over a duration of time. In variations, the output onto device850 can correspond to a controller device having one of various possibleform factors. The program 856 can configure the interface to graphicallyprovide gaming context and/or different user-interface paradigms forcontrolling the self-propelled device 810. The program 856 can operateto directly affect the content generated in these implementations basedon movement, position, or state of the self-propelled device 810.

In operation, the self-propelled device 810 can implement theprogrammatic runtime 816A using one or more sets of program instructionsstored in its program library 820. The program runtime 816A cancorrespond to, for example, a program selected by the user, or one thatis run by default or in response to some other condition or trigger.Among other functionality, the program runtime 816A can execute a set ofprogram-specific instructions that utilizes device functions and/orresources in order to: (i) interpret control input from the controllerdevice 850; (ii) control and/or state device movement based on theinterpretation of the input; and/or (iii) communicate information fromthe self-propelled device 810 to the controller device 850.

The program runtime 816A can implement drive control logic 831,including sensor control logic 821 and input control logic 823. Thesensor control logic 821 can interpret device sensor input 811 forcontrolling speed, direction, or other movement of the self-propelleddevice's drive system or assembly. The sensor input 811 can correspondto data such as provided from the accelerometer(s), magnetometer(s),and/or gyroscope(s) of the self-propelled device 810. The sensor datacan also include other information obtained on a device regarding thedevice's movement, position, state or operating conditions, includingGPS data, temperature data, etc. The program 816A can implementparameters, rules or instructions for interpreting sensor input 811 asdrive assembly control parameters 825. The input control logic 823 caninterpret control input 813 received from the controller device 850. Insome implementations, the logic 823 can interpret the input as acommand, in outputting drive assembly control parameters 825 that aredetermined from the input 813. The input drive logic 823 can also beprogram specific, so that the control input 813 and/or itsinterpretation are specific to the runtime program 816A. The driveassembly control logic can use the parameters, as generated throughsensor/input control logic 821, 823 to implement drive assemblycontrols.

In variations, the sensor/input control logic 821, 823 can be used tocontrol other aspects of the self-propelled device 810. In variations,the sensor/input control logic 821, 823 can execute runtime program 816Ainstructions to generate a state output 827 that can control a state ofthe device in response to some condition, such as user input or deviceoperation condition (e.g., the device comes to stop). For example, anillumination output (e.g., LED display out), audio output, or deviceoperational status (e.g., mode of operation and/or power state) can beaffected by the state output 827.

Additionally, the run time program 816A can generate an output interface826 for the self-propelled device program 856 running on the controllerdevice 850. The output interface 826 can generate the data thatcomprises feedback 818. For example, the output interface 826 cangenerate data that is based on position, movement (e.g., velocity,rotation), state (e.g., state of output devices), and/or orientationinformation (e.g., position and orientation of the device relative tothe initial reference frame). The output interface 826 can also generatedata that, for example, identifies events that are relevant to theruntime program 816A. For example, the output interface 826 can identifyevents such as the device being disrupted in its motion or otherwiseencountering a disruptive event. The output interface 826 can alsogenerate program specific output, based on, for example, instructions ofthe runtime program 816A. For example, the runtime program 816A canrequire a sensor reading that another program would not require. Theoutput interface 826 can implement instructions for obtaining the sensorreading in connection with other operations performed throughimplementation of the runtime program 816A.

Furthermore, the self-propelled device 810 can be operable in multiplemodes relative to the controller device 850. In a controlled mode, theself-propelled device 810 can be controlled in its movement and/or stateby control input 813 via control signals communicated from thecontroller device 850. In some implementations, the self-propelleddevice 810 can pair with the controller device 850 in a manner thataffects operations on the controller device 850 as to control orfeedback. The self-propelled device 810 can also be operable in anautonomous mode, where control parameters 825 are generatedprogrammatically on the device in response to, for example, sensor input811 and without need for control input 813. Further, the self-propelleddevice 810 can be operated along with the controller device 850 in a“touch and drive” mode, wherein the user can touch a selected locationon a video stream of the self-propelled device 810, and theself-propelled device 810 can autonomously maneuver to the correspondinglocation. Still further, in variations, the self-propelled device 810can act as a controller, either for the controller device 850 or foranother self-propelled device 810. For example, the device can move toaffect a state of the controller device 850. The device can operate inmultiple modes during one operating session. The mode of operation canbe determined by the runtime program 816A.

The self-propelled device 810 can include a library of instruction setsfor interpreting control input 813 from the controller device 850. Forexample, the self-propelled device can store instructions for multipleprograms, and the instructions for at least some of the programs caninclude counterpart programs that execute on the controller device 850.The library maintained on the self-propelled device can be dynamic, inthat the instructions stored can be added, deleted or modified. Forexample, a program stored on the self-propelled device can be added, oranother program can be modified.

When executed on the controller device 850, each program includesinstructions to recognize a particular set of inputs, and differentprograms can recognize different inputs. For example, a golf program canrecognize a swing motion on the controller device 850 as an input, whilethe same motion can be ignored by another program that is dedicated toproviding a virtual steering mechanism. When executed on theself-propelled device 810, each program can include instructions tointerpret or map the control input 813 associated with a particularrecognized input to a command and control parameter.

In variations, the self-propelled device can be able to dynamicallyreconfigure its program library. For example a program can be modified(e.g., through instructions received by the controller device 810) toprocess control input 813 that corresponds to a new recognized input. Asanother example, the self-propelled device 810 can be able to switchprograms while the self-propelled device 810 is in use. When programsare switched, a different set of inputs can be recognized, and/or eachinput can be interpreted differently on the self-propelled device 810.

Touch and Drive Method

FIG. 9 is a flow chart illustrating an example method of operating aself-propelled device. Referring to FIG. 9, a user of a controllerdevice (e.g., a smart phone, table, remote control, or other mobilecomputing device) can receive an image or a video feed from aself-propelled device. The self-propelled device need not be sphericalin nature, but rather can be any device capable of being controlled by acontroller device as described herein. The self-propelled device can beconfigured to include, among other features, a transparent sphericalhousing, a camera positioned within the housing, and a wireless linkwhich can stream the video feed to the controller device. In thismanner, the controller device can receive the feed from the camerapositioned within the self-propelled device (900). Further, thecontroller device can include a display screen, such that the feed canbe displayed on the controller device (902). The controller device caninclude touch screen features that allow the user to interact directlywith the displayed feed.

The user can then select an object or location point on the displayedvideo feed. Accordingly, the user selection of the location point isreceived by the controller device (904). The user can select thelocation point by touching an object on the screen corresponding to thephysical object in the self-propelled device's frame of reference.Additionally or alternatively, the user can provide voice commands thatidentify the object being displayed. Upon receiving the selection by theuser, the controller device can then generate a command signal based onposition information to the self-propelled device, causing it totraverse to the physical location point (908). The self-propelled devicecan traverse directly to the object, or perform maneuvers aroundobstacles to ultimately arrive at the object's location.

As an addition or alternative, the controller device can be furtherconfigured to determine a position of the object depicted on the displayscreen based on the user's and/or controller device's reference frame.The controller device can also map the relative position of the locationpoint displayed on the controller device with a position of the locationpoint in the self-propelled device's reference frame (906). In thismanner, the self-propelled device can be directed by the controllerdevice to traverse to the location of the object based on the mappedrelative positions via the image/video feed, wherein the controllerdevice can include processing means to calculate the relative positions.

CONCLUSION

One or more examples described herein provide that methods, techniquesand actions performed by a computing device are performedprogrammatically, or as a computer-implemented method. Programmaticallymeans through the use of code, or computer-executable instructions. Aprogrammatically performed step may or may not be automatic.

One or more examples described herein can be implemented usingprogrammatic modules or components. A programmatic module or componentmay include a program, a subroutine, a portion of a program, or asoftware component or a hardware component capable of performing one ormore stated tasks or functions. As used herein, a module or componentcan exist on a hardware component independently of other modules orcomponents. Alternatively, a module or component can be a shared elementor process of other modules, programs or machines.

Furthermore, one or more examples described herein may be implementedthrough the use of instructions that are executable by one or moreprocessors. These instructions may be carried on a computer-readablemedium. Machines shown or described with FIGs below provide examples ofprocessing resources and computer-readable mediums on which instructionsfor implementing examples of the invention can be carried and/orexecuted. In particular, the numerous machines shown with examples ofthe invention include processor(s) and various forms of memory forholding data and instructions. Examples of computer-readable mediumsinclude permanent memory storage devices, such as hard drives onpersonal computers or servers. Other examples of computer storagemediums include portable storage units (such as CD or DVD units), flashmemory (such as carried on many cell phones and tablets)), and magneticmemory. Computers, terminals, network enabled devices (e.g., mobiledevices such as cell phones) are all examples of machines and devicesthat utilize processors, memory and instructions stored oncomputer-readable mediums. Additionally, examples may be implemented inthe form of computer-programs, or a computer usable carrier mediumcapable of carrying such a program.

Although illustrative examples have been described in detail herein withreference to the accompanying drawings, variations to specific examplesand details are encompassed by this disclosure. It is intended that thescope of the invention is defined by the following claims and theirequivalents. Furthermore, it is contemplated that a particular featuredescribed, either individually or as part of an example, can be combinedwith other individually described features, or parts of other examples.Thus, absence of describing combinations should not preclude theinventor(s) from claiming rights to such combinations.

While certain examples of the inventions have been described above, itwill be understood that the examples described are by way of exampleonly. Accordingly, the inventions should not be limited based on thedescribed examples. Rather, the scope of the inventions described hereinshould only be limited in light of the claims that follow when taken inconjunction with the above description and accompanying drawings.

What is claimed is:
 1. A self-propelled device comprising: a wirelessinterface; a housing; a propulsion mechanism; a camera; and one or moreprocessors to execute an instruction set, causing the self-propelleddevice to: using the camera, generate a video feed; transmit the videofeed to a controller device via the wireless interface; receive an inputfrom the controller device indicating an object or location in the videofeed; and in response to the input, initiate an autonomous mode toautonomously operate the propulsion mechanism to propel theself-propelled device towards the object or location.
 2. Theself-propelled device of claim 1, wherein the self-propelled devicecomprises an aircraft.
 3. The self-propelled device of claim 2, whereinthe executed instruction set causes the self-propelled device toautonomously detect and maneuver around obstacles in propelling theself-propelled device towards the object or location.
 4. Theself-propelled device of claim 1, wherein the camera is mounted on agyroscope of the self-propelled device.
 5. The self-propelled device ofclaim 1, wherein the executed instruction set further causes theself-propelled device to: initiate a control mode to (i) receive commandinputs from the controller device, and (ii) operate the propulsionmechanism to accelerate and maneuver the self-propelled device based onthe command inputs.
 6. The self-propelled device of claim 1, furthercomprising: an inertial measurement unit; wherein the executedinstruction set further causes the self-propelled device to: receivefeedback from the inertial measurement unit to compensate for a dynamicinstability of the self-propelled device; and based on the feedback,dynamically stabilize pitch, roll, and yaw of the self-propelled device.7. The self-propelled device of claim 6, wherein the inertialmeasurement unit comprises at least one of a three-axis gyroscopicsensor, a three-axis accelerometer, or a three-axis magnetometer.
 8. Theself-propelled device of claim 6, wherein the propulsion mechanismcomprises a plurality of independent motors, and wherein the executedinstruction set causes the self-propelled device to dynamicallystabilize the pitch, roll, and yaw of the self-propelled device bygenerating correction signals for independent execution by the pluralityof independent motors.
 9. The self-propelled device of claim 6, whereinthe executed instruction set further causes the self-propelled deviceto: maintain an orientation awareness of the self-propelled device inrelation to an initial frame of reference.
 10. The self-propelled deviceof claim 1, further comprising: a global positioning system (GPS) deviceto provide GPS data to the controller device.
 11. The self-propelleddevice of claim 1, wherein the controller device comprises a mobilecomputing device executing a software application specific tocontrolling the self-propelled device.
 12. The self-propelled device ofclaim 11, wherein the mobile computing device comprises one of a smartphone or a tablet computer.
 13. A computer-implemented method foroperating a self-propelled device, the method being performed by one ormore processors of the self-propelled device and comprising: using acamera of the self-propelled device, generating a video feed;transmitting the video feed to a controller device via a wirelessinterface of the self-propelled device; receiving an input from thecontroller device indicating an object or location in the video feed;and in response to the input, initiating an autonomous mode toautonomously operate a propulsion mechanism of the self-propelled deviceto propel the self-propelled device towards the object or location. 14.The method of claim 13, wherein the self-propelled device comprises anaircraft.
 15. The method of claim 14, wherein in the autonomous mode,the one or more processors operate the propulsion mechanism toautonomously detect and maneuver around obstacles in propelling theself-propelled device towards the object or location.
 16. The method ofclaim 13, further comprising: initiating a control mode to (i) receivecommand inputs from the controller device, and (ii) operate thepropulsion mechanism to accelerate and maneuver the self-propelleddevice based on the command inputs.
 17. A controller device comprising:a display screen; one or more processors; and one or more memoryresources storing instructions that, when executed by the one or moreprocessors, cause the controller device to: receive a video feed from aself-propelled device; display content corresponding to the video feedon the display screen; receive a touch input on the display screenindicating an object or location in the video feed; in response to thetouch input, generate a command signal to command the self-propelleddevice to initiate an autonomous mode to autonomously accelerate andmaneuver towards the object or location; and transmit the command signalto the self-propelled device for execution.
 18. The controller device ofclaim 17, wherein the controller device comprises one of a smart phoneor a tablet computer executing a software application specific tocontrolling the self-propelled device.
 19. The controller device ofclaim 18, wherein the self-propelled device comprises aradio-controllable aircraft.
 20. The controller device of claim 17,wherein the executed instructions cause the controller device togenerate the command signal by (i) mapping a relative position of theobject or location in the video feed with a position of the object orlocation in a frame of reference of the self-propelled device, and (ii)including instructions to maneuver the self-propelled device toward tothe object or location based on the frame of reference of theself-propelled device.